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ABSTRACT 


The Aviation Selection Test Battery (ASTB) has been 


used to screen prospective aviation candidates for the Navy, 











Marines, and Coast Guard since World War : The Navy has 





continuously looked for ways to update and improve its 


aviation screening procedures and the latest supplement of 





the Performance-Based Measurement Battery (PBMB) currently 





under development is just one example. The Naval Operational 











Medicine Institute (NOMI) is currently investigating how 





well this test will assist with the difficult process of 
aviation candidate selection. This study was conducted to 


assess whether individuals with aviation experience would 





perform better on the PBMB than those with no aviation 
experience. Forty individual participated in this research, 


20 had formal aviation training. The results showed that 





experienced aviators performed significantly better on eye- 
hand coordination tracking tasks than the group with no 


aviation experience. 
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EXECUTIVE SUMMARY 


This thesis performed an assessment of the Navy’s 
Performance-Based Measurement Battery (PBMB) under 
development and intended to supplement the Aviation 
Selection Test Battery (ASTB). These findings will augment 


the validation study currently being conducted by the Naval 











Operational Medicine Institute (NOMI) in Pensacola, Florida. 








A total of 40 individuals attending the Naval 
Postgraduate School volunteered to participate in this 
study. Of those 40 volunteers, 20 had received formal 


aviation training as either a pilot or Naval Flight Officer 











(NFO). All participants were administered the test battery 


which consisted of three components: a direction orientation 








test, a dichotic listening test (DLT), and a multi-tracking 











task. Combinations of these components were then used to 





make up two multi-tasking tests: the multi-tracking with DLT 





subtests and the multi-tracking with the emergency 


procedures (EP) subtests. 


Results indicate that those with aviation experience 
performed better on all multi-tracking tasks. The aviation 
experience group and the group without aviation experience 
showed the largest significant difference on the multi- 
tasking test comprised of the multi-tracking and emergency 


procedures (F = 12.49, p = .001). Background demographics 





like the amount of video games played were determined to be 
immaterial. The aviation experience group and the group 


without aviation experience did not differ in regard to the 





performance on the direction orientation test (F = 0.57, p = 
48). 


x1il 


Overall, the PBMB was capable of detecting important 


eye-hand coordinated tracking skills and with further 








analysis and refinement to the scoring algorithm, this test 


battery should improve future aviation candidate selection. 
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I. INTRODUCTION 


A. BACKGROUND 


Since the armed forces of the United States started 


using aircraft in combat, there has been a dedicated effort 





to make certain that the right candidates are selected for a 








career in military aviation. The United States Navy (USN) 








quickly recognized that the individuals selected needed to 








possess the appropriate physical and mental attributes 





required to become successful aviators. Flying an aircraft 
has always been considered a dangerous and costly 


occupation. The costs required to properly train and equip 











a pilot or flight officer have continued to escalate. It is 





important that an organization have the ability to 


accurately select the right individual from a pool of 





applicants. In 1993, training a single Student Naval 


Aviator (SNA) in a jet aircraft exceeded $900,000 (Hewes, 





1994). Given that this is such an expensiv volution, the 





USN makes every attempt to minimize the chance of selecting 











an individual without the proper skill sets into the initial 





aviation candidate selection process. 


This monetary cost drives the Navy, Marine Corps, and 


Coast Guard services to keep flight school failure rates as 











low as possible. Improvement efforts resulted in the 











development of the Aviation Selection Test Battery (ASTB). 





Since the adoption of the ASTB, it has been consistently 





used as the criterion for selecting officer aviation program 








applicants. The ASTB was revised by the Naval Operational 








Medicine Institute (NOMT) in cooperation with the 


Educational Testing Services (ETS) in Princeton, New Jersey, 





in 1992 (NOMI, 28 Sep 2007). 








The United States, particularly NOMI, has continued to 





make improvements in the ASTB since its initial creation. 
Not all the proposed modifications have been formally 


adopted but those that made the cut have proven to improve 

















the selection process. Many of these tests wer created 
during the World War era and remain, for the most part, 
unchanged (Griffin, 1996). A prime example is the 


perceptual-cognitive “paper and pencil” type examinations. 


The ASTB has been a successful aptitude test in the 





candidate selection process. This test battery assesses an 





applicant’s math skills, the ability to extract meaning from 


written material, familiarity with mechanical concepts and 








simple machines, and the ability to perform mental rotations 





in order to determine the spatial orientation of aircraft in 





3-dimensional space. The ASTB also measures an individual’s 
knowledge of aviation and nautical terminology, familiarity 
with aircraft components and function, knowledge of basic 


aerodynamic principles, and the ability to grasp flight 








rules and regulations. From th ntire test battery, four 





scores ar derived from the combinations of the subtests. 





The predictive validity of the Academic Qualifications 
Rating (AOR) (used in prediction of SNA academic grades) is 
r= 0.45 (p < .001), while the validity of the Pilot Flight 
Aptitude Rating (PFAR) (used to predict SNA flight grades) 








is r = 0.35 (p < .001). The current test battery is viewed 


by many experts as adequate in its ability to predict 





individuals who will succeed during aviation training (NOMI, 


28 Sep 2007). 








Although the current test battery does have a proven 





track record, it could benefit from further improvement. One 





particular area under refinement is psychomotor testing. An 


appropriate eye-hand coordination test would be beneficial 





in refining the ability of the ASTB to predict whether or 








not a Student Naval Aviator (SNA) will succeed during the 





flight portion of aviation training. According to Damos 





(1996), the vast majority of current pilot selection 


batteries are better predictors of training performance 





rather than operational performance. Others share a similar 


belief on the present aviation test batteries. 





Additionally, Damos along with McFarland (1953) believe that 








data obtained from observational methods do not 
appropriately reflect some of the more important aspects of 


a pilot's job which are the cognitive and psychomotor skill 








sets (Damos, 1996). A task analysis performed by NAVAIR has 








shown that eye-hand coordination is one of the most 
important physical skills candidates should possess (Mangos, 


2005). 


The Federal Aviation Administration (FAA) also believes 





that psychomotor or physical skills are important traits for 











aviators. These types of skills are required to conduct many 
routine tasks when flying an airplane. Besides basic 


airmanship, some typical activities which involve eye-hand 











coordination include the ability to fly a precision 





instrument approach procedure, programming a GPS receiver, 


or using sophisticated maintenance equipment. As physical 








tasks and equipment become more complex, the requirement for 


integration of cognitive and physical skills increases at a 





proportional rate (FAA, 1999). 


B. PSYCHOMOTOR TESTING 


The idea of using eye-hand coordination testing in 


aviation selection is not a new concept. As early as the 





1930’s, tests involving psychomotor skills proved to be 














reasonably successful. During World War , the primary 





psychomotor test used for pilot selection was the Mashburn 
automatic serial-action complex coordinator. This test 
involved a timed coordination of stick and rudder movements 


in response to 40 different visual patterns and proved to be 





a good predictor of performance (DeHart, 2002). This 





apparatus did have a down side. It was bulky and special 


training was required to assemble and collect data. Because 








logistical requirements made it difficult to transport 





between testing sites, its use was discontinued by 1951 


(Deckart, 1988). 





Recent improvements in modern technology have allowed 
the Navy to directly address some of these limitations. The 
cost incurred in managing hardware, software, and technical 


support for such a large quantity of testing sites has 





become less expensive and more manageable. Additionally, 





the availability of the personal computer and off-the-shelf 
hardware such as joysticks, throttles, and rudder pedals has 


allowed psychomotor testing to become less cumbersome and 





easier to administer to applicants. Internet connectivity 
has provided an expanding reach in testing, scoring, and the 


maintenance of an accurate database on all prospective 





aviation candidates. These advancements have allowed the 
4 


aviation community to expand eye-hand coordination testing 





options and develop relevant tests to determine which 


particular tasks can help refine the aviation selection 





process. These advancements have resulted in approximately 
150 Navy testing facilities across the United States and on 
average 10,000 Navy, Marine, and Coast Guard applicants a 


year taking the ASTB (Olde and Walker, 2006). 


Since thes tests have becom asier to develop and 








administer, studies have been conducted as to their 





relevance. Delaney (1992) evaluated the use of a 





psychomotor task (PMT) in the aviation selection process. 








Included with this particular PMT was a dichotic listening 


task (DLT). The statistical evaluation of the automated DLT 














and PMT concluded that both skills contributed to the 








prediction of primary flight-training criteria (Delaney, 
1996). The results from this study suggested that assessing 
concurrent psychomotor tracking and DLT performance may be a 


particularly effective way of predicting performance in 





flight training. Large-scale validation studies of the Air 








Force's computerized psychomotor test have reinforced these 


findings (Carretta, 1989). 





Delaney (1992) determined that psycho-motor based 
measures could account for an additional 14.8% of unique 
variance, above what was already being accounting for by the 
ASTB. Griffin and Koonce (1996) had similar results and 


concluded that psychomotor based tests could explain 16% 





more variance than the standard paper and pencil version of 


the ASTB. These two studies indicate that psychomotor tests 





can explain approximately 15% more unique variability 


between aviators who get low flight grades versus those 


aviators who get high flight grades. This may be 





contributed to the fact that a psychomotor test measures 


eye-hand coordination and tracking skills, skills not 





currently assessed by the standard cognitive based version 


of the ASTB (Olde & Walker, 2006). 





The Navy’s Performance-Based Measurement Battery (PBMB) 
is a group of timed, interactive psycho-motor subtests 


assessing a number of different skills and abilities 








including multi-tasking, eye-hand coordinated tracking, task 
prioritization, decision making, and spatial orientation. 


These skills and abilities have been shown to be 





particularly important in aviation (FAA, 1999). Detailed 





directions are displayed on the computer screen prior to the 
commencement of all seven subtests. The PBMB also provides a 


practice session for all the subtests except the last one 





(an emergency procedures test). Also, all subtests had to be 
completed within the allocated time otherwise the 


participant would not receive a valid score. 


This thesis is intended to assess the performance of 
the current version of the PBMB, which is still under 
development and intended for use to supplement the Navy’s 


ASTB. These findings will augment the validation study 





being conducted by NOMI in Pensacola, Florida. Specific 


questions addressed in this thesis include: 








Do individuals with aviation experience perform better 





on the PBMB than those with no flight experience? 





Do individuals who play sports, video games, or have 
other unique demographic background characteristics perform 


better on the PBMB? 





Do participants have any usability issues when taking 


the PBMB? 


This project is in support of the Navy’s effort to 





develop a psychomotor test that can successfully identify 





applicants with good eye-hand coordination and multi-tasking 





abilities prior to being accepted into the aviation training 


pipeline. 
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II. METHODS 


Th latest version of the PBMB was administered to 40 





volunteers. This sample consisted of two subgroups. The 
first group was comprised of volunteer test subjects with 


formal aviation experience while the other group had no 





aviation experience. We hypothesize that the qualified 





aviators on average, possess more fine tuned psychomotor 





skills. We expected this group to average a higher score on 








the psychomotor test than the group with no aviation 





experience. It was beyond the scope of this study to 





investigate how the aviation group developed better eye-hand 
coordination skills. We simply assumed that they possessed 


those skills and that the battery would accurately assess 





this difference. We realize that these traits could be a 


manifestation of inherent natural ability or learned through 





countless hours of formal aviation training and experience. 





If humans have eye-hand coordination as an inherent trait, 
then it is anticipated that some of the participants with no 


formal flight training would score as well as those 





individuals with aviation experience, but as a group those 





with no formal flight training should produce a lower 


average score. The primary goal of this research is to 





explore the fidelity of this new selection test battery and 
investigate whether or not this particular version of the 


PBMB can correctly identify important characteristics such 





as eye-hand coordination and spatial orientation ability. 


A. SUBJECTS 


The PBMB was administered to 40 volunteers attending 
the Naval Postgraduate School located in Monterey California 
during a three week period in the Fall of 2007. All but one 


of the volunteers was serving on active duty and nine 





individuals were representatives from five different foreign 


military services. 





B. APPARATUS 


Advancements in computers and software technology have 
resulted in a substantial improvement in the way selection 
tests can be administered. They have also allowed for 
improvements in the level of sophistication and realism 
contained in current psychomotor tests. The version used 
during this research required applicants to use a joystick, 


throttle, and headphones along with other standard computer 





hardware to complet th test battery. All the hardware 








used is available for purchase off the shelf and the APEX 








delivery system allows for the test to be administered 





online. 


Participants were positioned in a quiet, comfortable 


environment at standard office desk and chair. The layout 





used for this experiment is depicted in Figure 1. The 





computer keyboard was required to complete the demographic 


section. Once the demographic information was filled out, 





the mouse and headphones wer used to perform the first 











subtest, direction orientation test (DOT). Upon completion 





of the DOT, the keyboard was set aside and a joystick and 
throttle were setup. The joystick, throttle, and headphones 


were used for the remainder of the examination. 


10 





Figure 1. Sitting environment used during the PBMB. 


This research project used a Thrustmaster HOTAS Cougar 


combination throttle and joystick, shown in Figure 2. These 





controls were designed to replicate those used in an F-16 
jet aircraft. For the purpose of standardization, the 


factory settings on both the joystick and throttle were 





maintained throughout the experiment, however participants 
could maneuver them into a comfortable position as long as 
the throttle was positioned on the left and the joystick was 


ither centered or on the right. 





11 





Figure 2. Thrustmaster HOTAS Cougar throttle and 
joystick combination used for the PBMB. 


Cc. PROCEDURES 


The PBMB consisted of the following subtests in the 


following order: 

















e Direction Orientation Test (DOT) 

e Dichotic Listening Test (DLT) 

e Vertical Tracking Test (VTT) 

° Airplane Tracking Test (ATT) 

e Vertical Tracking Test/Airplane Tracking Test 
(VIT/ATT) 

e VIT/ATT/DLT Multi-task Combination 

e VIT/ATT/Emergency Procedures (EP) Scenario 


Besides the direction orientation test, the PBMB was 
designed for each subsequent section to build on each other 


and provide practice with these novel tasks. The final 





composite score will be comprised of the DOT, the 


VIT/ATT/DLT, and the VTT/ATT/EP subtests. 





aby 


The ASTB PBMB contained a demographic section which was 


required to be completed befor 


not used in this 


softwar 


research. 





starting the examination but 
Questions included with the 


date of 





wer nam 


y 


birth, Social 





status, 


Point Average (GPA) 


Security 


prior military 


gender, race, place of birth, 


Number (SSN), present military 


service, educational level, Grade 





for highest completed education level, 


major, experience with flight simulator games or software, 


any formal flight training, any aviation training what 


level and number of hours. 


For the research conducted and contained in this 


thesis, all the participants were asked a different set of 





background and demographic questions designed specifically 


to gauge a participant’s eye-hand coordination. 


Additionally, any personal information such as a SSN was 





substituted with a subject number to assure anonymity. An 


example of the questionnaire is located in Appendix A. 





Direction Orientation Test (DOT) 





The direction orientation test is the first subtest of 











the PBMB. 





It consists of a series of timed exercises that 





requir th to determine aircraft’s 





subject an position 











relative to a target. 


This test is used in both the pilot 
and navigator selection process and assesses the applicant’s 
ability to orient their location from a map view to an 


outside aerial view of 





the ground. This particular skill is 





frequently used in aviation. 


For this subtest, the computer screen is divided into 





two sections. A tracker map is displayed on the left side 


of the screen which shows the location of the Unmanned 











Aerial Vehicle (UAV) and the direction is indicated by an 
arrow. North is always oriented at the top of the tracker 
map. The image that is displayed on the right side of the 
screen is the target. The target is shown as it is viewed 
through the UAVs camera, depicted below in Figure 3. This 
camera 1s mounted beneath an UAV and will always point 


straight ahead of the UAVs direction of flight. 





Camera View 





Tracker Map 
Za 





When you hear which parking lot to image you must move the mouse pointer 
over that parking lot and click the left mouse button. While North is always at the 
top of your Tracker Map, this is not always true for the Camera View. You must 
orient the Camera View with the top representing the direction the UAV is heading. 





Figure 3. Instruction page. Actual test does not 
display heading information or an arrow but it 
does show the same two camera views during the DOT 
test. 


The target used in this subtest consists of a building 


surrounded by four parking lots. One parking lot is located 





on each cardinal heading. The participant is asked to 
select one of the surrounding parking lots as quickly as 


possible. The specific targeting instructions are received 





verbally through headphones as well as written below the 


target map. The participant must move the mouse over the 
14 





parking lot to be chosen and then click the left mouse 





button when the arrow is displayed over that corresponding 








parking lot. The examinee will receive feedback on their 
selection. A green circle appears around the selected 
parking lot when correctly identified. The elapsed time 





required to make the decision is displayed inside the 














circle. If an incorrect parking lot is selected, a red 
circle will appear and the associated elapsed time is 
displayed inside that circle. 


The test was designed to provide a total of eight 
practice questions before the scored test began. By 
selecting the “previous” button upon completion of the 
eighth question, the practice session could be repeated as 


many times as desired. There are differences between th 











actual test and the practice test. During the practice 


session, if an incorrect target is selected, the correct 








parking lot will be displayed, this is not so during the 
actual test. The second difference is that after a practice 
question is answered, the participant must select the 





“continue” button to advance to the next question, but in 
the actual test the program advances automatically. The 
participants are told that their score for the direction 


orientation task will be calculated using a combination of 





speed and accuracy of the selected answers. 


Zi; Dichotic Listening Task (DLT) 


Pilots and flight officers often listen to multiple 





radio frequencies at the same time during a flight and it is 





imperative that they are able to focus on a particular 


callsign or other important radio transmissions. 
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During the DLT portion of the PBMB, the participant 
hears a series of numbers and letters presented to each ear 
through a pair of headphones, as displayed in Figure 4. The 
participant will be verbally asked to monitor a “target” ear 
for even or odd numbers. When an even number is heard in 
the “target” ear, the examinee is required to press the 
trigger of the joystick located in their right hand. When an 
odd number is presented in the “target” ear, the individual 
is required to press the thumb button of the throttle 
located in their left hand. 










Left Ear Right Ear 








Figure 4. Example of how the DLT test is presented to 
the subject. 





The DLT test is one minute forty seconds in length and 
scoring is determined by how quickly and accurately the 


participant responds to each number presented in the 





“target” ear. It is scored as incorrect if an individual 
responds to any number being presented to the non-target 


ear. 


3. Vertical Tracking Test (VTT) 


The vertical tracking test (VIT) takes one minute to 





complete and measures an applicant’s ability to track a 
moving target on a vertical axis. For this subtest, a 


yellow airplane and a red crosshair are displayed on the 
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left side of the computer screen. The yellow airplane moves 


up and down on the screen at increasingly faster speeds. 





The participant can only control the red crosshair. The 





object of the test is to keep the airplane inside the red 


crosshair for as long as possible. 





In order to move the crosshair up and down the screen, 





the participant uses the throttle controls located to their 





left. When the throttle is moved up, the crosshair will 








move up on the screen and conversely, moving the throttle 








down causes the crosshair to move down. If the participant 





is successful in targeting the airplane, the crosshair will 





turn green and when the targeting lock is lost, it returns 











to red. 
Three scores are recorded for this task. The first is 
a total number of correct responses. A correct response is 


recorded when the pixel distance between the crosshair and 








aircraft is within a programmed range at a predetermined 
time check during the task. The second score recorded is an 


error average distance. This distance is the average 





distance observed between the center point of the crosshair 
and the center point of the aircraft over the duration of 
the subtest. The third score is for the total number of 
redirects recorded during the task. A redirect is recorded 
when the subject maintains the crosshair on top of the 
aircraft for brief amount of time causing the aircraft to 


change course. 





During the VTT, as the participant correctly targets 


the airplane, it starts moving faster thus progressively 








increasing the level of difficulty. There are thr speeds 


with which the airplane can move and thus three levels of 
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difficulty. The VIT was designed to measure how many 


airplane redirects the participant can induce. 





4. Airplane Tracking Test (ATT) 


The fourth subtest in the PBMB is the airplane tracking 
test. This test takes one minute to complete and measures 
an individual’s ability to track a moving target in two 
dimensions. An airplane and red targeting crosshairs are 


displayed on the screen. The basis for this examination is 





for the participant to imagine that they are is attempting 








to target the airplane shown on the screen. The individual 


attempts to keep the crosshair centered on the airplane as 





it moves around the screen. The crosshairs will turn green 


when the airplane is successfully targeted. 








The examinee has no control over the speed or direction 
of the target. The test subject manipulates the crosshairs 
by moving the joystick located on their right. When the 
joystick is moved to the right the crosshairs move right. 


Conversely, when the participant moves the joystick to the 











left, the targeting crosshairs move to the left. When the 


joystick is pushed forward the crosshairs move up and when 





the joystick is pulled back the crosshairs move down. 





The scores recorded for the ATT are similar to the 





three described in the VTIT section. As the participant 








correctly targets the airplane, it starts moving faster thus 





progressively increasing the level of difficulty. The ATT 





was also designed to measure how many airplane redirects the 


participant can induce. 
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5. Airplane Tracking Test and Vertical Tracking Test 
(ATT /VTT) 





The ATT/VIT takes two minutes to complete and is the 
first PBMB subtest that combines two tasks. Both of the 


tasks required in this test have been previously performed 














individually. During this section, the subject is assessed 





on their ability to perform these tasks simultaneously. 


The examinee is required to use the joystick to target 





the airplane on the right side of the computer screen which 


is moving in two dimensions and operate the throttle control 





to target the airplane moving along the vertical axis on the 





left side of the screen, s Figure 5. 





LOCK ON 4% LOCK ON 1% Shots: 0 - Hits: 0 (0% 


Figure 5. Display used during the ATT/VTT. The 
aircraft and crosshair shown on the left are for 
the VIT subset. 
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The test subject must attend to each task equally 





during the ATT/VTIT because the scoring incorporates how well 








both airplanes are accurately targeted. 


6. VTIT/ATT/DLT Multi-task Combination 





This multi-task combination test takes three minutes to 


complete and simulates the high mental workload of some 





aviation tasks. This test is a mixture of three tasks 
previously performed. The VTT, ATT, and DLT are all 
administered during this subtest. The purpose of the multi- 


task combination test is to assess the subject’s eye-hand 


coordination while performing the dichotic listening test. 








The joystick and throttle operate in the same manner as 





the VIT/ATT subtest while the dichotic listening task is 


also performed. 





The participant must attend to all three tasks equally. 


Scoring for the multi-task combination subtest tracks the 





accuracy of targeting both airplanes, the number of 
redirects, and the accuracy and speed of the responses to 


the DLT. 





7. VTT/ATT/EP Scenario 


The emergency scenario test is intended to replicate 
the difficult task of dealing with problems while 
maintaining flight (a task pilots need to be able to 





perform). While flying an aircraft, emergencies are an ever 


present risk. Making inaccurate decisions and untimely 





responses can be very unforgiving. It is also important 





that the pilot continues to fly the airplane while dealing 


with an emergency scenario. 
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This two minute test taps an individual’s eye-hand 


coordination plus memory capacity. The emergency scenario 





test is the final subtest in the PMBM. It utilizes the 








vertical tracking test and the airplane tracking task while 





the participant responds to an emergency audio and visual 


warning. This warning will identify one of three emergency 





scenarios that are given during the subtest. 


The participant must carefully read the detailed 








instructions provided and memorize the associated responses 


for each emergency scenario. The applicant will be required 





to respond to all three emergency scenarios over the course 





of this subtest. An example of one emergency scenario is a 





verbal warning such as “Fire, Fire, Fire” followed by the 
illumination of a fire light. The test subject is then 
required to respond to the fire emergency using the 
responses described in the instructions. Fuel and power 
gauges are adjusted to the appropriate settings. The gauge 


displays are at the bottom of th screen, s Figure 6. 





Bach emergency contains three actions which can _ be 
accomplished by using the throttle located to the left. Two 


knobs and a button ar required to manipulate the 








appropriate gauges and set them to the correct position for 





the specific emergency displayed. If an incorrect response 





or no response is given, the screen will turn red. The 


applicant is then notified visually that the aircraft 





systems are operating under duress. If no response is given 





within th predetermined time, the screen will reset in 





preparation for the next emergency. 
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Fire 
EMERGENCY EMERGENCY EMERGENCY EMERGENCY EMERGENCY EMERGENCY 





Figure 6. The VIT/ATT/EP as viewed by the subject. 
Notic th fuel and power gauges added to the 
bottom right of the screen. 





The participant must continue to perform the VIT and 
the ATT tasks while dealing with the emergency scenario. 


All three tasks are considered in the final score for this 





subtest. 
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III. RESULTS 


Of the 40 volunteers, four were female and 36 male, the 





ages of the subjects ranged from 27 to 43 years old and the 


median and mean ages were both 33 years. 


For the purpose of this study, an individual with 





flight experience was defined as those participants who 
received any formal pilot or flight officer training, 20 met 


this general criterion. This group was comprised of two 





Naval Flight Officers (NFO) and 18 pilots qualified to fly 


jets, propeller aircraft or helicopters. The flight 








experience group also contained one participant with only a 


private pilot’s license (75 hours of flight time), a 





participant who started but did not complete flight school 
(200 hours flight time), and one participant that 





transitioned from a NFO to a pilot. The mean flight time 
was 1423.75 hours with the minimum number of hours being 75 
and 3000 being the maximum. One participant did have flight 


time as an observer but this gualification did not meet this 











study’s definition of flight experience and therefore they 





were not included in the flight experience group. 


Only two of the participants with aviation experience 


held a current instrument, instructor or Naval Air Training 





and Operating Procedures Standardization (NATOPS) 
qualification at the time of the experiment. The mean time 
since the last flight was 29 months. The longest time 


observed was 120 months and the shortest was seven months 


Since their last flight. 





Participants were asked how often they played video 


games and if any of those games were flight simulators. 
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Fifteen volunteers reported playing video games on a monthly 
basis and only two indicated they occasionally played flight 
simulator games. The mean time spent playing video games 


per month was 5.35 hours with the highest being 60 hours. 





All exploratory analysis conducted in this thesis was 





accomplished using JMP 7 and SPSS personal computer software 





packages. Unless otherwise noted all results are quoted at 











the two-tailed 0.05 significance level. 


A. SPATIAL ORIENTATION SKILLS 


1. Direction Orientation Test (DOT) 





A total of 48 questions were asked in the DOT subtest 
and each participant received two scores. One score was for 


the total number of correct responses and a second score for 





the total response time for all correct and incorrect 


answers. 


The mean score of correct responses for the aviation 
experience group was 39.95 with a standard deviation of 5.09 
while the no experience group had a mean of 38.15 and a 


standard deviation of 6.52. These two means were not 





statistically significant (F = 0.95, p= .34). 





The mean total response time observed for the group 





with aviation experience was two minutes twenty three 
seconds with a standard deviation of one minute thirteen 
seconds. The group with no experience took slightly longer 
with a mean time of two minutes thirty seconds and a 
standard deviation of one minute nineteen seconds, these 


means were not statistically significant (F = 0.04, p= .84) 
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A composite score was created to address the speed and 





accuracy trade-off of the DOT. The total number correct and 


total response time for each individual score were 





standardized and combined (weighted equally). The difference 





between the mean composite scores of the two groups (0.12 


for those with aviation experience and -0.12 for those 





without) was not statistically significant (F = 0.57, p = 
-46). 


B. LISTENING SKILLS 


1. Dichotic Listening Test (DLT) 


A total of 16 questions were asked during the one 





minute forty second DLT portion of the psychomotor test. 
Those with aviation experience had a mean number of correct 
responses of 11.9 and a standard deviation of 4.52 while 
those with no aviation experience had a mean number of 
correct responses of 10.15 and a standard deviation of 4.34 


(see Table 1). The means for the number of correct 








responses in the DLT were not statistically significant (F = 


1.56, p = .22). 





The mean total response time for correct responses 
observed for the group with aviation experience was 0:16.11 
seconds with a standard deviation of 0:06.12 seconds. The 
group with no experience had a mean response time of 0:15.94 
seconds and a standard deviation of 0:07.83 seconds. An 
average time for a correct response was calculated by 


dividing the total correct time by the total number of 





correct responses. The number correct and average time wer 


then standardized and weighted equally to calculate a 





composite score. The difference between the mean composite 
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scores of the two groups (0.19 for those with aviation 
experience and -0.19 for those without) was not 


statistically significant (F = 3.178, p = .083). 





Total Response Time 






































Adjusted Number for Correct 
Number Correct Correct Responses 
EXPERIENCE 11.9 4.52 20.83 7.906 00:16.1 00:06.1 
INO EXPERIENCE 10.515 4.34 | LT. 16 7.602 | 0015.9 00:07.8 | 
Table 1. Summary of both adjusted, unadjusted response 








times for the DLT subtest. 


Cc. EYE-HAND COORDINATED TRACKING SKILLS 


1. VTT, ATT, and VIT/ATT Tests 





For the purpose of this study, w defined eye-hand 
coordinated tracking skills as the one and two dimensional 
tracking accomplished with the joystick and throttle. The 
subtests containing tracking skills varied in duration. The 


shortest time was one minute for the VIT and ATT with the 





longest being three minutes for the VTIT/ATT/DLT. 


For each of the VTT and ATT tracking skills, a total of 





three scores were recorded. The first score was the number 
of correct responses, the second was the average distance 
error between the crosshairs and the airplane, and the third 


score was the total number of times the participant caused 





the aircraft to redirect because of capture. It should be 








noted that the lower the error average distance the closer 


the crosshairs remained to the aircraft during the test. 
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The correlations between the different tracking tasks 





and the different tracking measures wer all highly 
correlated (see Table 2). The first three rows in each 


matrix are the number correct, error average, and number of 





redirects for the VTT. Rows three through six are the 


number correct, error average, and number of redirects for 





the ATT test. Row seven is a sum of the number correct for 
the VTIT and ATT test. Row eight is an average of the error 
average for the VIT and ATT test and row nine is the sum of 





the total number of redirects for the VIT and ATT test in 
each subtest. Since the number correct, error average, and 
number of redirects are so highly correlated, only the 
number of redirects will be reported from this point on 


(number of redirects was selected becaus th PBMB was 





designed to use it as its primary measure of tracking 


skill). 
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VIT ATT Total 
VIT Error VIT ATT Error ATT Total Error Total 
Correct Avg Redirect Correct Avg Redirect Correct Avg Redirect 

















Correct 1.00 

Error Avg -0.93 1.00 

Redirect O99: =—04:92 

Correct 0.71 -0.62 

Error Avg -0.60 0.61 

[ Redirect 0.70 -0.62 1.00 

Total correct 0.93: =0.85 0.91 1.00 

Total Error Avg -0.84 0.88 -0.86 -0.92 1.00 
Total Redirect 0.91 -0.83 0.93 1.00 -0.92 

















VIT ATT Total 
VIT Error VTIT ATT Error ATT Total Error Total 
Correct Avg Redirect Correct Avg Redirect Correct Avg Redirect 

Correct 1.00 
Error Avg -0.94 1.00 
Redirect 099°  =0::95 
Correct 0.68 -0O.71 
Error Avg -0.70 0.69 
T Redirect 0.68 -0O.71 1.00 
Total Correct 0.94 -0.92 0.88 1.00 
otal Error Avg -0.86 0.89 -0.87 -0.94 1.00 
Total Redirect 0.92 -0.92 0.91 0.99 -0.94 












































Table 2. Correlation matrix of scores for the 
VIT/ATT/DLT and VIT/ATT/EP. 





2. Vertical Tracking Test (VTT) 





The xperienced group induced redirects at a 


Significantly higher rate than those without aviation 





experience, 17.9 and 15.5, respectively, F(1,39) = 4.59, p= 





-04 (see Table 3). 


3. Airplane Tracking Test (ATT) 





Again, the experienced group induced redirects at a 


Significantly higher rate than those without aviation 





experience, 9.95 and 6.05, respectively, F(1,39) = 7.92, p= 
.008 (see Table 3). 
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4. Vertical Tracking Test/Airplane Tracking Test 
(VTT/ATT) 


The total number of redirects for the VTIT/ATT subtest 





was calculated by adding the VTT number of redirects with 


the ATT number of redirects. Th xperienced group induced 











redirects at a significantly higher rate than those without 
aviation experience, 26.55 and 18.5, respectively, F(1,39) = 


9.72, p= .004 (see Table 3). 


5. VTIT/ATT/DLT Multi-task Combination 





In this subtest, the VIT/ATT scores were combined as in 


the previous section. The number of correct responses from 








the DLT was also recorded. 





Th xperienced group induced redirects during the 





VIT/ATT/DLT subtest at a significantly higher rate than 





those without aviation experience, S335 and 2. 3-6 1-5 


respectively, F(1,39) = 10.66, p = .002. However, the DLT 





mean number correct was not statistically different between 














the group with aviation experience and those without 





aviation experience, Zig correct and 20.65 correct 
respectively, F(1,39) = 0.06, p= .81 (see Table 3). When a 


composition score for speed and accuracy was calculated 





using the average time for a correct response (same method 








described in the DLT section), the DLT mean composite scores 





between the two groups were not significant (F = 0.11, p= 


e193) 





To determine the impact of the DLT on tracking skills, 





the mean number of redirects was adjusted for the one minute 





time differenc between the VIT/ATT subtest and the 











VIT/ATT/DLT subtest so these two tests could be directly 
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compared (see Table 3). Both groups experienced a drop in 


the mean number of redirects when the additional DLT task 





was added. The adjusted mean number of redirects for the 


VIT/ATT was compared to the mean number of redirects for the 








VIT/ATT/DLT for each group. Assuming unequal variance, the 


means for the group with aviation experience were 39.83 and 








37.35 (Prob. > |t| 0.59) and the means for the group without 





aviation experience were 27.75 and 23.15 (Prob. > |t| 0.25). 


Although these means are not statistically different, they 





do trend in the expected direction. 





The adjusted DLT mean number of correct responses (see 





Table 1) was compared to the mean number of correct 


responses for the DLT section of the VTT/ATT/DLT for each 








group. Assuming unequal variance, the means for the group 





with aviation experience were 20.83 and 21.1 (Prob. > |t| 


0.89) and the means for the group without aviation 





xperience were 17.76 and 20.65 (Prob. > |t| 0.22). These 


means are also not statistically different, however it is 











interesting to note that both groups performed better on the 








DLT the second time they took it (VTT/ATT/DLT) even though 





they were also performing a dual-tracking task. 


6. VTIT/ATT/EP Scenario 


The emergency scenario subtest was two minutes long and 





this section was treated similar to the VTIT/ATT/DLT. The 





VIT/ATT number of redirects for the VTIT/ATT/EP subtest was 


combined in a single score. The number of correct emergency 





responses was tallied with a maximum score achievable being 
three. The elapsed time to respond to each emergency was 


also recorded. 
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Th xperienced group induced redirects during the 


VIT/ATT/EP subtest at a significantly higher rate than those 





without aviation experience, 21.05 and 11.9, respectively, 
F(1,39) = 12.49, p = .001 (see Table 3). However, the mean 
number of correct EP’s was 2.2 for the group with experience 


and 1.85 without aviation experience, these means were not 








Statistically different (F = 1.12, p =  .30). When a 
composite score taking into account the speed and accuracy 


of responses during the emergency scenario was calculated 








(using the same method described in the DLT section), the 


composite mean scores (0.40 for the group with experience 














and -0.40 for those without xperience) between the two 
groups were not statistically significant (F = 2.05, p = 
716) 


Since both the VIT/ATT and VIT/ATT/EP subtests were two 
minutes long, no adjustment to the mean number of redirects 


was necessary to determin th impact of tracking while 





performing the EP task (see Table 3). Both groups 
experienced a drop in the mean number of redirects. 
Assuming unequal variance, the means for the number of 
redirects of the aviation experience group were 39.83 and 


31.58 (Prob. > |t| 0.049) and the means for the group 





without aviation experience were 27.75 and 17.85 (Prob. > 


lt| 0.011). This mean difference is statistically different 





and trended in the expected direction for both groups. 
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Table 3. 


STD DE 
3.06 
ST 





STD DE 
4.37 
4.39 





Number Redirects 


STD DE 
8.16 
8.17 





Number Redirects 


Mean STD DEV Mean STD DEV STD DEV 


16.26 
10.67 


Number Redirects 


STD DE 
8.95 
7.34 













Adjusted Number 
Number Redirects Redirects 


| Mean STD DEV| 
ee 
46.5 


Adjusted Number 
Number Redirects Redirects 


| Mean STD DEV| 
— 
93.6 


Mean 
39.83 
27.15 


Sls 
231.5 


Adjusted Number 
Redirects 


Mean 
31.58 
17.85 





Adjusted Number 
Redirects 







Adjusted Number 
Redirect 


STD DE 
9.18 
11.9 








STD DE 
6.56 
Deg deal! 













STD DEV 
12.24 
12.26 








# DLT Correct 





16.26 
10.67 


2141) 
20.65 


4.54 
7.01 














EP Correct 
STD _DEV 
13.42 242 0.77 
11.02 1.85 1.27 


























DLT Average Time 
STD DEV 


Mean 
1,34 
1253 





Mean 
28.4 
23%:9 





EP for the tracking skill subtests. 











0.49 
0.61 


EP Average Time 


STD DE 
4.5 
12:8 





Summary of both the unadjusted and adjusted 


scores for the number of redirects, DLT and 








D. PARTICIPANT DEMOGRAPHICS 


Prior to taking the PBMB, participants were asked a 





series of questions about their perceived eye-hand 


coordination skills and multitasking abilities. These 


responses along with the participants’ age, gender, aviation 


experience, amount of video games played, ability to juggle, 


and ability to dribble a basket ball were analyzed using a 
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stepwise regression to predict the total number of redirects 
for the VTIT/ATT/EP subtest. Only aviation experience was 


Significant (p = .001). When the same predictors were used 








to model the number of redirects for the VTT/ATT/DLT 
subtest, aviation experience (p = .004) and gender (p = 


.047) were statistically significant predictors. Note, there 








were only four females in the study and only one had flight 





xperienc this small sample size and unegual cells 


preclude any predictions based on gender. 
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IV. DISCUSSION 


Certain subsets of the psychomotor test did support our 
hypothesis that individuals having flight experience would 
on average, score higher than those without flight 


experience. 


The most significant results were the mean scores for 





the eye-hand coordinated tracking tasks. These skills 
included tracking two airplanes with a throttle and 


joystick. While we focused on the total number of 





redirects, either the total number correct responses or the 





average error distance would have detected the difference in 





tracking ability between th xperienced and non-experienced 








groups. It has been claimed that eye-hand coordination and 
tracking skills are important in the field of aviation 


(Mangos, 2005) and these findings suggest that our selected 





group of aviator not only had these skills but that the PBMB 








was able to detect them. 








The VIT, ATT, VIT/ATT, VIT/ATT/DLT, and the VIT/ATT/EP 
sections all produced significant results when comparing the 


mean number of redirects between the two groups. 


The aviation experience group and the group without 











aviation experience showed the largest significant 
difference when performing the VTT/ATT/EP (F = 12.49, p = 
001). When the adjusted mean number of redirects for 


VIT/ATT/ subtask were compared to the adjusted mean number 
of redirects for the VIT/ATT/EP subtask, both groups had a 
significant decrease in mean scores. This drop in 


performance was expected because participants were required 
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to manipulate thumb wheels located on the throttle control 





during the emergency scenario which diverted attention away 





from aircraft tracking. 





The VIT/ATT/DLT subtest (although not as robust as the 














VIT/ATT/EP) resulted in a significant difference between th 


aviators and non-aviators (F = 10.66, p = .002) and was 








consistent with the findings of Delaney (1992). The 





differenc between the adjusted mean redirects for the 











VIT/ATT subtest and the mean redirects for the VTIT/ATT/DLT 


subtest showed a decline in performance. Again, this trend 





was expected due to participants performing another test. 





When the adjusted means for the one minute DLT were 








compared to the DLT portion of the VTT/ATT/DLT subtask, we 








anticipated that the mean of the average time per correct 





response for both groups would be higher when the 
participant’s attention was divided among tasks. However, 
the results showed that both groups mean average time for a 


correct respons decreased. We observed a marginal 





improvement in th xperienced group’s performance from a 





mean of 1.34 to 1.30 seconds and a more substantial 





improvement for the group with no experience from 1.53 to 


1.26 seconds. Although the observed decrease for the two 














groups was not statistically different, this improvement was 


unexpected. The most logical explanation for this 








improvement can be attributed to the novelty of the DLT task 














causing participants trouble. However, by the time they took 
the VIT/ATT/DLT section, they may have figured out the 


procedure and thus their scores improved. This explanation 
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is even more likely given the participants comments about 
misunderstanding during the practice portion of the DLT 
subtest (see the Observations and Recommendations section 


below). 








The DOT did not distinguish between th xperienced 





aviators and the non-experienced group. Since the Navy 


currently uses a paper and pencil style spatial abilities 





test in the ASTB which is different from the DOT spatial 











ability test, they may have to continue relying on that 











test. The instructions for the DOT test state that the 





scoring will be determined by the speed and accuracy of a 
response. Two scores were provided in the data output for 


this section. One score was a total accumulated time for 





correct and incorrect responses and the other reported th 
total number of correct responses. Finding the ideal 
weighting of these scores in a composite score is beyond the 
scope of this paper, however we did look at a composite 
score based on an equally weighted standardize version of 
the number correct and total response time and found no 


Significant difference between the mean scores for the two 











groups. The PBMB DOT test was unable to detect a difference 





in spatial abilities between those with flight experience 





and those without. Using a more sophisticated weighting 


scheme may prove more fruitful. 





The DLT was another subtest of the PBMB that did not 








distinguish between the group with aviation experience and 


the group with no aviation experience. When the mean number 





correct responses for the DLT was examined, there was no 


difference between groups. Although the composite score of 





speed and accuracy did not differ between groups there was 
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an increase in the F statistic, which was F = 1.56 for the 
mean number correct and F = 3.18 for the composite score. 
The composite score was based on an equally weighted 


standardize version of the number correct and total time for 





correct responses. A more sophisticated model of weighting 
the number correct and total time for correct responses 


might prove a more robust measure. 





It is apparent that those with aviation experience did 





perform better than those without on the eye-hand tracking 
skills contained in the psychomotor test. Either of the 
last two subtests can adequately measure this difference. 


The DOT subtest did not support our hypothesis, but it 














appears that dichotic listening may have affected 
performance during the VTT/ATT/DLT subtest. These two 
sections may benefit from improvements in data collection, 





and scoring, and further evaluation of how the final scoring 


model is produced. 


A. OBSERVATIONS AND RECOMMENDATIONS 


The directions included befor ach subtest were found 








to have sufficient detail. One interesting observation to 


note pertained to the emergency procedure section. 





Participants were told to adjust fuel and power levels to 





either high, low, or neutral depending on the emergency they 





were presented. However, the gauges were displayed with a 
red stripe located at the top, green in the middle, and 


yellow at the bottom, as shown in Figure 6. Although one 








would normally assume that the top of gauge is a high 





setting, it had a red indication which may be interpreted as 
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being low. This subtle difference in directions has the 








potential to cause some confusion and should be clarified in 











future test versions. 


All practice secessions preceding each subtest could 


only be performed once except for the direction orientation 








test. A few of the volunteers were observed repeating the 





DOT practice session multiple times. This was made possible 


by selecting the “previous” button upon completion of the 





eighth practice question. It is unknown whether this extra 








practice had any influence on the results for the DOT 








section. The directions should clearly state that 





participants can repeat the practice section of the DOT or 





the possibility for repeated practice should be eliminated. 








The practice session for the DLT is different than the 


full DLT test. The 30 second practice session does not 





7 





switch “target” ears while the actual test does require th 








participant to shift attention from one ear to the other 
ear. This inconsistency may have caused some confusion. An 


individual may not have been expecting a switch in “target” 








ears on the DLT, but by the end of the test they usually 





understood the task. This initial shock of switching ears 








may explain why both groups increase their DLT score later 





during the VTT/ATT/DLT subtest. Consideration should be 





given to changing the DLT practice session to include a 


switch in “target” ears. Furthermore, increasing the length 





of the actual DLT may ensure sufficient time for learning 


this novel task. 


Participants were given an opportunity to provide 





feedback on the psychomotor test when the questionnaire was 


completed. One issue mentioned was that the factory 
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sensitivity settings on the joystick and throttle were 
awkward. The throttle had to be moved forward or backward 
approximately 2/3 of the way to get the crosshairs to begin 


to move. Also, full right or left movement of the joystick 








was difficult to achieve without the control unintentionally 
moving on the table top. Some refinement to the sensitivity 


setting may be desirable. 


Another comment from the majority of the volunteers was 





the overall difficulty of the test. Of the 40 participants, 








38 of them felt that the test was challenging while 24 found 


ie heO> ab xtremely challenging. Only two volunteers 








considered it to be about the right level of difficulty. 


40 


V. RECOMMENDATIONS FOR FURTHER RESEARCH 


The scope of this study was limited and we only touched 





the surface of the required analysis needed prior to 
fielding the Navy’s new psychomotor test for aviator 


selection. There exists a multitude of opportunities for 





further research and subtl refinements in the current 


version of the PBMB. 


The first area that should b researched further 





involves th development and analysis of a model to 





correctly weight response time with correct responses for 


both DLT and DOT tests. In this study we elected to equally 














weight these scores which may not be ideal. Exploring a 





better weighting scheme should be addressed. 


We elected to only look at subjects who had already 





completed some type of formal flight training. Valuable 











insight was gained into the tests ability to recognize eye- 





hand coordination and tracking skills but it is still 
unknown whether or not this was a learned skill or an 
inherent ability. Furthermore, the aviator’s adage of 
aviate, navigate, and communicate may have played a role 
their disparate scores in the multi-tracking task and 


emergency scenario subtests. 





None of our test subjects were under the age of 27 or 


currently selected for aviation and awaiting formal flight 





training. Further research should involve testing this 


target population. These individuals should be administered 





the psychomotor test prior to having any aviation 





xperience. They should be tracked throughout their formal 


flight training and upon completion or discontinuation from 
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flight school. Their flight school performance could be 
used to design an appropriate weighted model for the PBMB. 
Results from a study of this nature should provide further 


insight into the PBMBs predictive capability and eventually 





allow for a standard measure to be established For 


acceptance into flight school. 
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APPENDIX 








Questionnaire 
Section 1 
1. Participant Number: 
2 Age 
3 Sex 
a. Male 
by Female 
4. On a scale of 1 to 5 with 5 being the highest, how 
would you rate your overall hand eye coordination? 
3} —- Excellent 
4 —- Above average 
S) —- Average 
2 —- Below Average 
1 —- Poor 
ote Do you have any aviation experience? 
a. Yes 
lon No 





If you answered no to question 5 skip to question 13. 
6. Is your flight experience as a: 

Pilot 

Flight officer 

Flight engineer 

Crewmember 











Qaago ow 





ds Is your aviation time 
Military 
Civilian 

Ci. Both 





oO w 


8. What type of airframe (circle all that apply)? 
Fixed wing - Jet 

Fixed wing - propeller 

Helicopter 

Other 
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OF Approximately how many total hours would you say you 
have (rounding is fine)? 











10. What is the highest qualification you have held? 























(Instructor, mission commander, test pilot etc): 
11. Are any of your qualifications still current? 

a. Yes - all 

b. Yes -— some 

b. No 
12. How long has it been since your last flight (in 
months) ? 
13. On average, how much time per month do you play video 


games (in hours)? 








14. Are any of the video games flight simulators? 
a. Yes 
b. No 


15. On a scale of 1 to 5 with 5 being the highest, circle 
how athletic you think you are 


i) —- Should have turned pro 

4 —- Above average 

3 — Average 

2 —- Below average 

1 - Always the last kid picked 


16. Are you currently playing any sport (recreational or 
organized) ? 





a. Yes 
by No 
17. Have you ever played any organized sport? 
a. Yes 
b. No 
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18. What sport and what position (example: baseball, short 























stop) ? 
19. On a scale of to 5 with 5 being the highest, circle 
how well would you think you are at multitasking? 

3) —- Excellent 

4 —- Above average 

3 — Average 

2 —- Below Average 

1 - Poor 
20. The game is on the line and it all comes down to one 
final play. If the play is made, your team wins, if not, 
you lose. Do you want to be the one who: 

a. Is in position to make the play. You have the 


utmost confidence in your ability. 
bi Is in the play but the entire thing does not hinge 


on your actions. 














om Is out there to back up the individual up who 
makes the game winning play. 

da. Is not currently in the game but has a front row 
seat. 

es Not applicable - wouldn’t be playing in a sporting 
event 





21. Can you juggle? 
a. Yes 
b. No 


22. How well would you say you can dribble a basketball? 





—- Excellent 


— Above average 


—- Below Average 


5 
4 
3 — Average 
2 
1 


= "POOr 
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Section 2 











ae How well do you think you did on the test? 
iS) Excellent 
4 —- Above average 
3 —- Average 
2 —- Below Average 
1 —- Poor 
Dis How would you rate the difficulty of the exam? 
5 —- Extremely challenging 
4 - Challenging 
3 -— Just about right 
2 —- Walk in the park 
1 - I could do it with my eyes closed 
Ss What did you like about the exam? 
4. What did you not like about the exam? 








Would you say this is an accurate assessment of your hand- 
eye coordination (explain why or why not)? 
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